The phrase "lunch downtown" means having a meal in a restaurant or cafe located in the central part of a city during the midday hours.